An Approach for Hybrid-Memory Scaling Columnar In-Memory Databases

In-memory DBMS enable high query performance by keeping data in main memory instead of only using it as a buffer. A crucial enabler for this approach has been the major drop of DRAM prices in the market. However, storing large data sets in main memory DBMS is much more expensive than in disk-based systems because of three reasons. First, the price for DRAM per gigabyte is higher than the price for disk. Second, DRAM is a major cause for high energy consumption. Third, the maximum amount of DRAM within one server is limited to a few terabytes, which makes it necessary to distribute larger databases over multiple server nodes. This so called Scale-Out approach is a root cause for increasing total costs of ownership and introduces additional overhead to an in-memory database system landscape. Recent developments in the area of hardware technology have brought up memory solutions, known as Storage Class Memory, that are capable of offering much higher data density and reduced energy consumption than traditional DRAM. Of course these advantages come at a price: higher read and write latencies than DRAM. Hence, such solutions might not replace DRAM but can be thought of as an additional memory-tier. As those solutions are not available on the market today, we investigate the usage of an available bridge-technology combining high-end flash SSD and DRAM referred to as hybrid-memory. In this paper we present an approach for columnar inmemory DBMS to leverage hybrid-memory by vertically partitioning data depending on its access frequency. We outline in what way our approach is suited to scale columnar in-memory DBMS to overcome existing drawbacks of ScaleOut solutions for certain workloads.